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ABSTRACT 


'"^This  paper  is  concerned  with  the  optimal  control  of  discrete-time 
linear  systems  that  possess  randomly  jumping  parameters  described  by  finite 
state  Markov  processes.  For  problems  having  quadratic  costs  and  perfect 
observations,  the  ootimai  control  laws  and  expected  costs-to-go  can  be 
precomputed  from  a  set  of  coupled  Riccati-like  matrix  difference  equations. 
Necessary  and  sufficient  conditions  are  derived  for  the  existence  of 
optimal  constant  control  laws  which  stabilize  the  controlled  system  as  the 
time  horizon  becomes  infinite,  with  finite  optimal  expected  cost. 
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1. 


>n  and  Prt 


•ormulation 


Consider  the  discrete-time  jump  linear  svsti 


*k+l  *  Ak(rk)xk+  WV 


^k+l^'V”  *  pk+l(i’j) 


k  *  k  1 1  * i jN  * 

o’ 


where  the  initial  state  is 


X(V  *  x0  ’ 


r<V  *  r0 


Here  the  x-process  is  n-dimensional ,  the  control  u  6  R  ,  and  the  form 
process  <rk:k*Kg,  .  .  .  ,N>  is  a  finite-state  Markov  chain  taking  values  in 
M  *  <1,  2,  .  .  M>,  with  transition  probabili ties  p^fi,?). 

The  cost  criterion  to  be  minimized  is 


WV  s  E 


/  IuWVUk  +  x/k*:Qk+l(rk+l)xk+l] 

k=k7 


+  X  NKT(rN)XN 


The  matrices  RjJj),  Q^+1  < 3 ) *  and  K^j)  are  posi tive-semidef ini te  for  each  j 


and  k.  In  addition,  we  assume  that 


Rk(j)  +  B'k<j) 


'  pk+l(j’i)(Wn3 


Bk(j)  >  0 


The  role  of  this  condition  will  become  clear  in  the  sequel.  Note  in 
particular  that  (5)  is  satisfied  if  Rk<j)  >  0  and  Gk(j)  >,  0  for  all  j  6  M 
at  all  times  k . 

This  kind  of  problem  formulation  can  be  used  to  represent  the  control 
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of  systems  subject  to  abrupt  phenomena  such  as  component  and 
interconnection  failures.  We  call  this  the  lump  linear  Quadratic  (JIQ) 
control  oroblem.  The  continuous-time  version  of  this  oroblem  was 
apparently  first  formulated  and  solved  by  Krasovskii  and  Lidskii  [2].  The 
problem  was  studied  later  by  Wonham  [3].  He  obtained  sufficient  conditions 
for  the  existence  and  uniqueness  of  solutions  in  the  JLQ  case,  and  also 
derived  a  separation  theorem  under  Gaussian  noise  assumptions  for  JLQ 
control  problems  with  Markovian  forms  and  noisy  x  (but  perfect  r) 
observations.  Sworder  [4]  obtains  similar  results  using  a  stochastic 
maximum  principle  and  has  published  a  number  of  extensions  with  his  co¬ 
workers,  including  [4]  -  [9].  Stochastic  minimum  principle  formulations 
for  continuous  time  problems  involving  jump  process  have  also  been 
considered  by  Rishel  [10]  Kushner  [14],  and  others.  Robinson  and  Sworder 
[11,12]  have  derived  the  appropriate  nonlinear  partial  differential 
equation  for  continuous-time  jump  parameter  systems  having  state  and 
control-dependent  rates.  A  similar  result  appears  m  the  work  of  Kushner 
and  an  approximation  method  for  the  solution  of  such  problems  has  been 
developed  by  Kushner  and  DiMasi  [13]. 

Discrete-time  ve-sions  of  the  JLQ-contrei  problem  have  not  been 
thoroughly  investigated  in  the  literature.  A  special  case  of  the  x- 
independent  JLQ  discrete-time  problem  is  considered  in  Birdwell  [15-17], 
and  the  finite-time  horicon  x-independent  problem  is  solved  in  Blair  and 
Sworder  [16].  Minor  extensions  are  discussed  in  [17],  In  this  paper  we 
develop  necessary  and  sufficient  conditions  for  the  existence  of  steady- 
state  optimal  controllers  for  the  discrete  time  JLQ  problem.  These 
conditions  are  much  more  complicated  than  in  the  usual  discrete-time  linear 
quadratic  regulator  problem.  Specifically,  these  conditions  must  account 
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for  the  difference  in  the  stability  properties  of  the  closed  loop  system 
for  different  values  of  r For  example,  it  is  possible  for  a  particular 
component  of  x  to  diverge  when  r^  takes  on  a  particular  value,  if  r^ 
takes  on  this  value  rarely  enough  and  if  this  component  of  x  is 
stabilized  sufficiently  when  the  system  is  in  other  structural  forms.  Thus 
one  finds  that 

.  stable  closed-loop  dynamics  in  each  or  all  of  the  structural  forms 
is  neither  necessary  nor  sufficient 

.  stabilizabili ty  of  the  dynamics  in  each  form  is  neither  necessary 
nor  sufficient 

.  controllability  of  the  dynamics  in  each  form  is  neither  necessary 
nor  sufficient 

for  the  existence  of  steady-state  optimal  controllers  yielding  finite 
excected  cost. 

In  the  next  section  we  review  the  basic  form  of  the  solution  to  the 
discrete-time  JLQ  problem  over  a  finite  time  horizon  and  in  Section  3  we 
present  examples  that  illustrate  several  jualitative  features  of  the 
solution.  In  Section  4  we  oresent  the  rather  complicated  necessary  and 
sufficient  conditions  for  the  existence  of  a  steady-state  solution  for 
time-invariant  JLQ  problems  over  infinite  horizons,  and  in  Section  5  we 
present  an  example  illustrating  this  condition  and  several  other  examoles 
which  serve  to  show  that  simpler  conditions  such  as  stabilizabilitv  or 
controllabili ty  are  neither  necessary  nor  sufficient.  Section  6  contains 
simpler  sufficient  conditions  for  the  existence  of  solutions  in  the 
infinite  horizon  case,  and  Section  7  contains  a  brief  summary. 
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Tht  optimal  control  law  can  be  derived  using  dynamic  programming.  Let 
',Vxk*rk)  th*  #xP*cttd  cost-to-go  from  state  (:<k,rk)  at  the  t i me  k 
(after  x'kQ(rk)xk  **  charged)! 

WV  *  X,NWXN 


yk‘xk’rk3  =  min  E 


u/kftk(rk)uk  +  x/k+ll3k+l(rk+l)xk+l 
+  yK+l<rk+l’*k+l> 


(5) 


Proposition  1:  Consider  the  discrete-time  noiseless  Markovian-form  jump 

linear  Quadratic  optimal  control  problem  (1)  -  (4).  The  optimal  control 
law  is  given  bv 

u.  ,  =  -l.  ,  (j)  x.  ,  for  r.  ,  *  i  e  M 

K-l  k-1  k-1  k-1  '  — 

k  =  kg,  kQ+l . N 

where  for  each  possible  form  j  the  optimal  gain  is  given  bv 

WJ)  *  "WV  +  ,s' 

where 

*  JL 

0  k(j)  *  \  pkO,i)  [Qk(i)  +  Kk(i)}  (?) 

i=l 

Hence  the  sequence  of  sets  of  positive  semi-def ini te  symmetric  matrices 
<Kfc_i ( j ) s  j  6  M>  satisfies  the  set  of  H  coupled  matrix  difference  equations 

Wj>  -  AWJ)0V”  -  Bk-ifj)  Wj)3  (3) 
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with  terminal  conditions 


KN(j)  *  KT(j). 

The  value  of  the  optimal  axpactad  cost  (3)  that  is  achieved  with  this 
control  law  is  given  by 

x'.K,  I 

0  ko  0  0 

The  proof  of  this  result  appears  in  [1]  and  is  sketched  in  the  appendix. 
An  earlier  and  essentially  identical  result  was  established  in  (16], 

Note  that  the  j  6  M>  and  optimal  gains  j  6  M>  can  be 

recursively  computed  off-line,  using  the  M  coupled  difference  equations 
The  M  coupled  Riccati-like  matrix  difference  equations  cannot  be 
written  as  a  single  nM-aimensional  Riccati-equation . 


.i  t 
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3.  Examples 

In  this  section  some  qualitative  aspects  of  the  JLQ  controller  given 
in  Proposition  1  are  illustrated  via  a  time-invariant  scalar  example  with 
M*2  forms.  This  example  serves  to  point  out  issues  that  arise  in  the 
consideration  of  steady-state  JLQ  controllers  in  the  following  sections. 


b,  u. 

if  r,  *  1 

1  k 

k 

+ 

b2uk 

if  %  =  1 

i » J )  sPxj 


min  E 


+  “\R<V  1  +*JNKT<rN> 

k^O 


(9) 


In  this  case  the  cost  matrix  sequences  ( j ) ,  j  6  M>  may  or  may  not 
converge  as  k  decreases  from  N,  and  furthermore,  x^  may  or  may  not  be 
driven  to  zero,  as  shown  in  the  following. 

Example  1 ;  Consider  the  following  choice  of  parameters  for  (9): 


k+1  3  xk  +  uk 

if  r 

k+1  =2xk  +  2uk 

p.  .  *  .5,  K_(j)  »  0,  Q(j)  =  1,  Pi))  *  1  for  j  *  1,2 

1  I  J  ! 

The  optimal  costs,  control  gains  and  closed-loop  dynamics  are  given  in 
Table  1,  for  four  iterations. 

As  the  table  indicates,  in  this  case  the  optimal  costs  and  gains 
converge  quickly.  Furthermore,  note  that  in  the  ’worst  case"  of  *  2  for 
all  k, 

lim  |x  I  l  lim  (.S)1^"1  | x. i  «  0. 

N — >oo  N — >oo 

Thus  x  is  driven  to  zero  bv  the  optimal  controller. 
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This  example  demonstrates  the  ‘passive  hedging"  behavior  of  the 
optimal  controller.  That  is,  possible  future  form  changes  and  their 
associated  costs  are  taken  into  account.  To  see  this,  consider  the  usual 
LQ  regulator  gains  and  cost  parameters  (as  if  PusP22al  Pl2a°21a°*’ 

which  are  listed  in  Table  2. 

Comparing  Tables  1  and  2,  we  note  that  for  k  <,  N-2  the  gains  of  the 
Proposition  1  JLQ  controller  are  modified  (relative  to  LQ  controller)  to 
reflect  future  form  changes  and  costs.  The  JLQ  controller  has  higher  r=l 
gains  to  compensate  for  the  possibility  that  the  system  might  shift  to  the 
more  expensive  form  r®2.  Similarly,  the  r*2  gains  are  lower  in  the  JLQ 
controller  reflecting  the  likelihood  of  future  shifts  to  r^*l.  I 

Example  2;  Here  we  choose  the  parameters  of  (9)  so  that  the  optimal 
closed-loop  systems  in  different  forms  are  not  all  stable,  although  the 
expected  value  of  x  is  driven  to  zero.  Let 


*k+l  a  xk 

+  uk 

if 

II 

JSC 

V- 

1 

*k=l  *  2x 

k  +  uk 

if 

Tk  = 

2 

°11  =  P21 

=  .3 

p12 

=  p_„  =  .1 

where 

vj 

>  *  0, 

Q(  j) 

=  1 

'  =1,2 

R(l) 

=  1, 

R(2) 

=  1000 

Thus  there  is  a  high  penalty  on  control  in  form  2. 

This  system  is  much  more  likely  to  be  in  r*l  than  in  r*2  at  any  time. 
We  might  expect  that  the  optimal  control  strategy  may  tolerate  instability 
while  in  the  expensive-to-control  form  r*2,  since  the  system  is  likely  to 
return  soon  to  the  form  r  *  1  where  control  costs  are  much  less. 
Computation  for  four  iterations  demonstr ates  this,  as  shown  in  Tables  3 
and  4. 
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As  our  analysis  in  subsequent  sections  will  confirm,  these  quantities 
converge  as  (N-k) — >oo  .  Note  that  the  closed-loop  system  is  unstable 
while  in  r* 2. 

Direct  calculation  of  the  expected  value  of  x^,  given  xQ  and  rQ,  shows 
that  IE  (xk>|  decreases  as  k  increases.  This  is  shown  in  Table  3.  In 
four  time  steps,  E<x>  is  reduced  by  over  95%  if  initially  the  system  is  in 
form  1  and  6&%  if  it  starts  in  form  2.  Note  that  if  the  system  starts  in 
the  expensi ve-to-control  form  r=2,  x  is  allowed  to  increase  for  one  time 
steo  (until  control  while  in  r  =  1  is  likely  to  reduce  it).  j 
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Kk(l).Lk(l) 

Kk(2).Lk(2) 

VbiLka) 

a2-o2lk(2) 

k=N-l 

.5 

.3 

.5 

.4 

k*N-2 

.  623 

.363 

.377 

.263 

k=N-3 

.636 

.975 

.364 

.251 

k=N-4 

.637 

.875 

.363 

.243 

Table  1: 

Optimal  Cost 
dynamics  for 

and  Controller 
Example  1. 

Parameters, 

and  closed-loop 

Kk(D  =  Lk(l) 

V2)  =  Lk(2) 

(with  *  1) 

(with  p22  *  i) 

k=N-l 

.5 

.3 

k=N-2 

.  6 

.878 

k  =N=3 

.615 

.383 

k=N-4 

.618 

.383 

Table  2s 

Standard  LO  Solution 

for  Example  1 . 

Vn 

K.  (2) 
k 

V1' 

Lk<2) 

k=N 

0 

0 

— 

— 

k*N-l 

.5 

3.396 

.5 

1.398xl0“3 

k*N-2 

.649 

7.385 

.649 

3.672xl0‘3 

k»N-3 

.699 

9.269 

.699 

4.603xl0"3 

k=N-4 

.719 

10.198 

.718 

5.060xl0'3 

’’’able  3s 

Optimal  gains 

and  costs  of 

Example  2. 
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Pi 


k*N-l 


.5 


arbiLk(1)  vb2Lk(2) 

1.998 

k*N-2  .359  1.996 

k=N-3  .301  1.995 

k=N-4  .281  1.995 

Table  4:  Closed-loop  optimal  dynamics  of  Example  2 


-1 

O 

II 

if  r0  *  : 

X 

o 

1.0 

1.0 

X1 

.231 

1.995 

E<x2> 

.132 

.938 

E<x3> 

.069 

.491 

E<x.> 

.045 

.319 

Table  5:  Elx^}  f°r  Example  2. 
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We  now  consider  the  control  problem  in  the  time-invariant  case  as  the 


time  horizon  (N-kg)  becomes  infinite.  Specifically  consider  the  model 
(1)  i  (2)  with  *k(rk)  *  A(rK),  Bk(rk)  =  8(rk)  and  *  pi  J  •  w*  wish 
to  determine  the  feedback  control  law  to  minimize 


I  N-l 


I 


1 


1  lm 


El  \  Cu/kR(rk)uk  +  x/k+iQ(fk+l)xk+l]  +  x/NKTlrN''XN||x0’r0  j  (1° 


k*k. 


(N-k  )  —  >oo 


For  future  reference,  from  Proposition  1  the  optimal  closed-loop 
dynamics  in  each  form  j  6  M  are 


Vi =  Ws 


where 


Dk<j)  =  Afj) 


fll) 


where  Q  is  defined  in  (?)  (in  the  time-invari ant  case,  of  course, 

k 


only  KM’)  in  (?)  may  vary  with  k). 


Before  stating  the  main  result  of  this  section,  we  recall  the 
following  terminology  pertaining  to  finite-state  Markov  chains: 

.  A  state  is  transient  if  a  return  to  it  is  not  guaranteed. 

.  A  state  i  is  recurrent  if  an  eventual  return  to  i  is  guaranteed. 


State  i  is  accessible  from  state  j  if  it  is  possible  to  begin  in 
j  and  arrive  in  i  in  some  finite  number  of  steps. 


States  i  and  j  are  said  to  communicate  if  each  is  accessible  from 
the  other. 


A  communicating  class  is  closed  if  there  are  no  possible 
transitions  from  inside  the  class  to  anv  state  outside  of  it. 


A  closed  communicating  class  containing  only  one  member,  j, 
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.  A  Markov  chain  state  set  can  be  divided  into  disjoint  sets  T, 
C.,...,C  where  all  of  the  states  in  T  are  transient,  and  each 
is  a  closed  communicating  class  of  recurrent  states. 

Define  the  cover  C*  of  a  form  j  6  M  to  be  the  set  of  all  forms 
accessible  from  j  in  one  time  step.  That  is, 

C*.  -  <i  6  Ms  p(j,i)  *  0>. 

The  main  result  of  this  section  is  the  following: 

Proposition  2:  For  the  time-invariant  Markovian  JLQ  problem  the  conditions 
described  below  are  necessary  and  suf f i ci en t  for  the  solution  of  the  set  of 
coupled  matrix  difference  equations  (6)-(8)  to  converge  to  a  constant 
steady-state  set 

<K(  j)  >.0:  j  6  M> 

as  (N-k 0 ) — >oo.  In  this  case  the  K(j)  are  given  by  the  M  coupled 
equations 

K( j )  *  A'( j)Q*( j )Df j)  (12) 

★  ★ 

where  D(j)  is  defined  as  in  (11)  with  0  ^(j)  replaced  bv  0  (j).  In 
turn,  0*(j)  is  defined  in  (?)  with  K^(j)  replaced  bv  K.(j);  that  is 

*  -Ji- 

Q  (j)  *  \  p  . [Q< 1 )  +  K(i)3  (13) 

/ 

i=0 

Furthermore  the  steady  gains  L(j)  in  the  steadv-state  optimal  control  law 

u(rk’V  =  "UV*k  (!4) 

are  given  by 

L.  *  [ R ( j )  +  B'( j)Q*( j)B(j)]"1  B ' ( ; ) Q* ( j )  A(j)  (15) 

Thus  under  the  conditions  described  below  the  optimal  infinite  horizon  cost 


Discrete  Time  Markovian  JLQ  Optimal  Control 


Page  12 


The  conditions  to  be  satisfied  art  as  follows.  There  exists  a  set  of 
constant  control  laws 

*  -F(j)xfc  j  *  1,.  •  * |M  (16) 

so  that 
Condition  1: 

For  each  closed  communicating  class.  ,  the  expected  cost-to-go  from  (x^  = 
x,  r.  *  j  6  C.)  at  time  k  remains  finite  as  (N-k) — >oo.  This  will  be 

k  l 

true  if  and  only  if  for  each  closed  communicating  class  C.  ,  for  all  forms 
j  6  C^,  there  exists  a  set  of  finite  positive  semi-definite  n  x  n 

matrices  <  Zj,  Z2>  ...,Z,Ci|  )  satisfying  the  ICJ  coupled  equations 


-32. 
v  P 
/_ 
t=0 


J  j 


[A.  -  B . F  ] ' r  <0.  +  F.'  R  F  >[A.  -  S  F .  J 1 
J  J  )  3  3  3  3  3  3  3 


Z,  =  I 


oo 

\ 

/ _ 

t=l 


t-1 


3  3 


[A.  - 


W'*! 


\  p  z 
/  73  3 


I  q  6  C 
t  q  *  3 


(17) 


Note  that  in  the  case  of  an  absorbing  form  j  (ie.,  a  singleton 
communicating  class)  Z^  reverts  to  the  quantity 


-22  .  . 
Z.  «  \  [A.  -B.F .]'*  <Q  +  F'  R.F  >[A.  -  B.F.]1 

3  /  1  J  I  3  3  3  3  3  3  3 

t-0 


Once  we  are  in  an  absorbing  form  our  problem  reduces  to  a  standard  LQ 
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problem  and  Condition  1  in  effect  states  that  unstable  modes  in  such  a  form 


that  lead  to  nonzero  costs  must  be  controllable. 

Condition  2: 

For  each  transient  form  j  e  I  [  H,  the  expected  cost-to-go  is 
finite.  This  is  true  if  and  only  if  set  of  finite  positive  semi-definite 
n  x  n  matrices  <  G^Gj,...,  G|T)  >  satisfying  the  |T|  coupled  equations 


G.  = 


oo 

\ 

/_ 

t«0 


p  [A. 
*33  3 


B  F  .  ] '  *  <Q.  +  F.'  R.F  >[A. 
3  3  3  3  3  3  3 


B  F  ] 
3  3 


/_ 

t=l 


t-1 
P33 


t*  - 


3 


B  F  ] '  i  \  p .  Q  + 
3  3  ,  7 j*  3  q  q 


I  q  6  I 
I  4  *  3 
l_ 


\ 

/w 

q  6M-T 
<3  *”)” 


°3q<:'q 


tA.  - 


B  F  ] 
3  3 


I 


_l 

(13) 


Condition  1  states  that  it  is  possible  to  achieve  finite  expected  cost 
after  the  form  process  leaves  the  set  of  transient  states  and  enters  one  of 
the  closed  communicating  classes.  Note  that  for  absorbing  states  (i.e. 
1C. I  =1),  Condition  1  reduces  to  the  usual  LG  condition.  Condition  2  states 
that  the  expected  cost  from  any  transient  form  is  finite.  This  precludes 
the  possibility  of  an  unstable  mode  of  x^  growing  without  bound  in  mean 
square  either  leading  to  infinite  accrued  cost  while  the  form  resides  in 
the  transient  state  set  (this  occurs  if  the  x^  mode  is  observable  through 
the  cost  in  transient  forms)  or  to  infinite  cost  once  the  form  jumps  into 
a  closed  communicating  class  (if  this  mode  becomes  observable  after  the 
transition).  0 
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The  proof  of  the  proposition,  which  is  given  in  [1],  is  quits 
straightforward,  and  wa  confine  ourselves  here  to  sketching  the  basic  idea. 
Necessity  is  clear,  since  if  conditions  1  and  2  are  not  satisfied  for  any 
control  law  of  the  type  (16)  then  the  finite  horizon  optimal  control  laws 
cannot  converge  to  one  with  finite  cost  as  (N-kg) — >oo  .  To  show 
sufficiency,  one  first  shows  that  if  one  applies  the  control  law  (16), 
then,  under  conditions  1  and  2,  the  expected  cost  is  finite  as  (N-k^)-->oo. 
In  fact  it  is  given  by 

x/(k0)2(r(k0))x(k(J)  if  r(kfl)  6  M  or  T 

x ' ( k g )  Q(r(k0))x(k  )  if  r(k  )  6T 

This  establishes  an  upper  bound  on  the  optimal  cost  matrices  (j)  for  the 
finite  time  horizon  problem  for  the  particular  case  when  the  terminal  costs 
K. . ( j )  *  0.  Furthermore,  in  this  case  the  K.  (j)  are  monotone  increasing  as 
(N-kg)  increases,  and  thus  they  converge.  It  is  then  immediate  that  the 
limi ts 

lim  K  (j)  »  K( j ) 

KO 

( N-k  Q )  —  >00 

satisfy  (16).  Straightforward  adaptations  of  standard  LQ  arguments  then 
allow  us  first  to  extend  the  convergence  result  to  the  case  of  arbitrary 
terminal  cost  matrices  for  the  finite  horizon  problem  and,  secondly,  to 
show  that  there  is  a  unique  set  of  positive  definite  solutions  of  (16). 
Conditions  1  and  2  of  Proposition  2  take  into  account 

.  The  probability  of  being  in  forms  that  have  unstable  closed  loop 
dynamics 

.  The  relative  expansion  and  contraction  effects  of 
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unstable  and  stable  form  dynamics,  and  how  the  closed-loop 
eigenvectors  of  accessible  forms  are  "aligned*.  That  is,  it  is  not 
necessary  or  sufficient  for  all  (or  even  any)  of  closed-loop  dynamics 
corresponding  to  sufficient  forms  to  be  stable,  since  the  interaction  of 
different  form  dynamics  determines  the  behavior  of  ECx'^x^l. 

These  various  characteristics  will  be  illustrated  in  the  examples  in 
the  next  section.  The  Conditions  in  Proposition  2  differ  from  those  of  the 
usual  discrete-time  linear  quadratic  regulator  problem  in  that  necessary 
and  sufficient  Conditions  1,  2  replace  the  sufficient  condition  that  the 

(single  form)  system  is  stabilizable.  Unfor tunately  these  conditions  are 
not  easily  verified.  There  is  no  evident  algebraic  test  for  (17), (19)  like 
the  controllabili ty  and  observability  tests  in  the  LQ  problem.  The  use  of 
the  conditions  in  Proposition  2  will  be  demonstrated  in  the  examples  that 
follow. 

It  is  important  to  note  that  even  if  the  conditions  of  Proposition  2 

are  satisfied,  we  are  not  guaranteed  that  x^ - >0  in  mean  souare.  One 

obvious  reason  for  this  is  that  Conditions  1  and  2  are  trivially  satisfied 

(with  F  ( j ) ,  Z(j),  G(j)  all  zero)  if  Qij)  =  0  in  all  forms.  Of  course,  the 

same  comment  applies  in  the  usual  linear-quadratic  problem.  In  that  case,  a 

set  of  conditions  that  guarantee  that  x^ — >0  in  mean  souare  are  the 

stabilizabili ty  condition  mentioned  previously  and  the  requirement  that 
1/2 

(A,Q  )  be  detectable. 

Sxynply  3: 

One  might  conjecture,  given  the  LQ  result,  that  Conditions  1  and  2 

1/2 

together  with  the  requirement  that  (A(j),Q  (j))  be  detectable  for  each  j 

might  be  sufficient  for  the  JLQ  problem.  This  is  not  the  case,  however,  as 
one  can  certainly  construct  deterministically-jumping  systems  (i.e.  time- 
varying  linear  systems)  which  are  counterexamples, such  as  the  following. 
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A(l)  * 


8(1)  *  0 


:  0  2 

s  1/2  0 


Q(l)  *  :  1  0  s 

:  0  0  s 


A(2) 


•  • 
01/2:  Q(2)  ■  :  0  0 

2  0:  :  0  1 


8(2)  *  0 


'12  *  '21  *  1 

The  following  corollary  presents  one  sufficient  condition  that  guarantees 
that  *k~”>0  in  mean  sQuar** 

Corollary  1:  Consider  the  time-invariant  JLQ  problem,  and  suppose  that  the 
Conditions  1  and  2  of  Proposition  2  are  satisfied.  Supoose  also  that  tne 
closed  loop  transition  matrix  A( j )-B(; >L( j )  is  invertible  for  all 

"/‘x  kxk' - >0  the  matrix  Q(  j  )  +  L(j>i  is  positive 

for  at  least  one  form  in  each  closed  communicating  class. 

Before  sketching  the  proof  of  the  corollary  it  is  worth  providing  an 
example  that  illustrates  the  types  of  situations  that  motivated  the 
inclusion  of  the  assumption  that  A(  j  )-B(  j  )LC.t )  :s  invertible  for  all 


; .  Then 
def ini te 


Cxamole  4; 

Consider  a  scalar  system  with  form  dynamics  illustrated  in  Figure  1  where 
Ad)  =  2  ,  A(2)  *  0  ,  A(3)  *  1 

8(1)  =  8(2)  =  8(3)  *  0 
Q ( 1 )  =  Q(2)  «  0,  QC3)  *  1 

In  this  case,  assuming  that  the  initial  form  is  211  ->  ix  IS  not  difficult 
to  show  that  ECxk23  —  >oo,  while  the  cost  incurred  over  the  infinite  horizon 
is  zero,  even  though  Q(3)  *  1.  The  reason  for  this  is  that  the  form  process 
is  likely  to  remain  in  form  1  for  too  long  a  time,  cut  this  large  value  of 
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the  state  is  not  penalized  because  of  tha  nulling  of  tha  state  at  tha  time 

of  tha  first  transition  to  form  2.  Nota  also  that  in  this  case,  although 

E[xk2]  divargas,  xfc— >oo  with  probability  1.  I 

For  simplicity  in  our  proof  of  tha  corollary,  iat  us  assuma  that  there 

is  a  single  closed  communicating  class.  Tha  extension  to  several  classes 

is  straightforward.  First  let  us  denote  by  j*  tha  form  specified  in  the 

Corollary;  i.e.,  j*  is  in  tha  closed  communicating  class,  and 

8  .  [Qij*)  +  L(j*)/R(j*)L( j*)I  *  *  >0  '19) 

min 

where  8  .  (A)  •  smallest  singular  value  of  A. 

min 

Nota  next  that  if  we  apply  tha  optimal  steady-state  control  law  as 
specified  in  Proposition  2,  and  if  rk*3 >  then  the  cost  accrued  at  time  k  is 
x'  [Q(  j )  +  L(i)'  R(J)  10)1 

Suppose  that  <t.>  is  any  seauence  of  strictly  increasing  stopping  times  so 
that  r^.  *  j*.  Than  under  the  conditions  of  Proposition  2,  the  optimal  cost 
J*  is  finite,  and  in  fact: 

*  -22 

oo  >  J  *  E  \  x'k[Q(rk)  +  L'(rk)R(rk)L(rk)]xk 
k=0 

-22 

>  E  \  x' .[Q<j*>  +  L'<j*)R(j*>L<3*)]x,. 

y  \  1  ^  1 

i»0 

-22  2 
l  U  \  E  i  lix  irj 

/_ 
i*0 

From  this  we  can  immediately  conclude  that 

lim  E  [  I  lx  1 12  3  *  0 

i — >oo  1 


( 20) 


(21) 
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What  wa  wish  to  show  is  that 


lim  E  t  llx.irj 

k  -->oo 


and  we  do  this  by  contradiction .  Specifically  suppose  that  (22)  is  not 
true:  that  is,  we  can  find  an  6  so  that  for  any  positive  integer  m,  there 
exists  another  integer  K(6,m)>i  m  so  that 

E[  "*K(6,n„"2  1  >  6  <23) 

We  will  show  that  this  supposition  contradicts  (21)  bv  constructing  a 

sequence  of  stopping  time  for  which  (21)  does  not  hold  if  (23)  does.  Let 

t  *  The  earliest  time  after  K(6,0)  that  the  form  process 
0  is  in  state  j* 

a  The  earliest  time  after  both  K(S,k)  and  t(S,k-l) 
that  the  process  is  in  state  j* 

Denote  bv  'J  the  set  of  form  trajectories  that  beqin  in  state  m  and  end  in 
m 

state  3+  without  any  intermediate  visits  to  j*.  For  anv  U  6  U  ,  let 
0(u)  denote  the  closed-loop  state  transition  matrix  along  the  trajectory 
u .  Then 


E:  1  !*tk'  !’3  *  E  JE  C  1  ^  xtk  1  '  1  XK(  e  ,k )  ’  rk  8  rn’  j 


■  E  IE  t  IU(u)  *K(#tk)ll  I  *K(e,k)  ’  rk  8  m]  j 


*  E  |  xK(e,k)  E  1  9#fV  •(uk)  1  rk  8  m  1  xK(e,k )  j 


I  (24) 


where  uk  denotes  the  form  trajectory  from  K(e,k)  to  t^.  Note  that  the 
invertibili ty  assumption  immediately  implies  that 
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V, 


*m  *  'min  <  E  [  9'(uk)  •'V  1  rk  m  ]>  >  0 
Letting 


g  •  min  g 

HI 


we  see  that  (23)and  (24)  together  imply  that 
E  [ 1 1  xtkl |21  >  M  6 
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5.  Example 

The  following  simple  scalar  example  illustrates  the  conditions  of 
Proposition  2. 

Example  5:  Consider  the  form  dynamics  depicted  in  Figure  2, where  the  x- 
process  dynamics  are  autonomous  in  all  forms: 

xk+l  =  a(rk)xk  rk  6  <1*2«3*4«5»6»7> 

and  Q(j)  >  0,  V  j,  Here  6  is  an  absorbing  form,  <3,4>  is  a  closed 

communicating  class,  and  T  *  <1,2,5,7>  is  the  set  of  transient  forms.  For 

the  absorbing  form  r  =  6,  condition  1  yields 

(i)  a2(6Xl, 

and  in  this  case 

Z:'S)  =  0(6) 

l-a*S'€) 

For  the  closed  communicating  ciass  <3,4>,  (17)  gives  the  coupled  equations 
Z<3)  =  0(3)  +  3~(3)Z(4) 

Z(4)  =  Q( 4)  +  a2( 4)Z(3) 

Consequently 


Z(3)  =  - = - g .  CQ( 3)  +  a“( 4)  0(4)] 

1  -  a  (3)a“(4) 


Z ( 4 )  =  — = - 5 .  [ Q(4)  +  a2(3)  0(3)) 

l-at3)  a  (4) 


Thus  for  Z3,  Z4  to  be  positive  (as  in  Condition  1)  we  must  have 
(ii)  a2(3)  a2(4>  :  1 

(i.e.  the  two-step  dynamics  corresponding  to  the  form  transitions  3-4-3  or 
4-3-4  must  be  stable).  For  the  transient  forms  <1,2,5,7>,  (18)  yields 
6(1)  *  0(1)  +  a2(l)  G( 2) 
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G(2)  *  Q(2)  +  a  (2)  Cp21Q(1 ) 3  +  P23  Z(3)  +  p2g  2(6) 


-22 

G(5)  *  Q(5)+  \  p 
/_ 
t=l 


t-1  2t 


55 


a  ( 5 )  [  0(5)d55  +  p53  2(3)  ] 


oo 


G(7)  *  Q(7)  +  \  P77t_1  a2t(7)  [  Q(7)p,7  +  p?2G(2)  ] 

t^l 


From  the  equations  for  G(l)  and  G(2), 


G(l)  = 


Q(l)  +  a2(l)G(2)  +  Cp23Z<3)  +  p2g  2(6)  )  a2(l) 
1  -  a2( 1)  a2( 2)  p21 


G(2)  = 


Q(2)  +  a£(2)Q(l)p21  +  P23Z(3)  +  p2g  2(6) 


1  -  a2(l)  a2(2)  o 


21 


So  for  0  <  G(l),  G(2)  <  oo  we  have 


(lii)  a2(l)  a2(2)  p?1  <  1. 


From  the  expression  for  G(5)  we  see  that  for  0  <  G(5)  <  oo  we  have 


( i  v) 

with  the  resulting 


o55  a  (5)  <  1 


G(5) 


0(5)  +  o53  a(5)  Z(3) 

1  -  P55  s2(5> 


From  the  expression  for  G(7)  we  see  that  for  0  <  G(7)  <  oo  we  have 


(v)  P7?a2(7)  <  1 

with 


[  0(7)  +  p,  a2( 7)  Q(2)  ] 
G(7) . --r . . 


1  -  P77 
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The  condition*  (i)-(v)  above  result  from  the  necessary  and  sufficient 


conditions  of  Proposition  2,  applied  to  this  oroblem.  For  this  example  we 
see  that 


The  absorbing  form  (r*6)  must  have  stable  dynamics;  (1) 

one  of  the  forms  in  the  closed  communicating  class  <3,4> 
can  be  unstable  as  long  as  the  other  form's  dynamics  make 
up  for  the  instability;  (:i) 

transient  forms  r  *  5,7  can  have  unstable  dynamics  as  long 
as  the  probability  of  staying  in  them  for  any  length  of 
time  is  low  enough;  (iii),(v) 

some  instability  of  the  dynamics  of  forms  r  =  1,2  is  okav 
so  long  as  the  probability  of  repeating  a  2 — >1 — >2  cycle 
is  low  enough; ( i  v ) . 


3 

In  the  oroof  of  the  LG  problem,  the  existence  of  an  upper  bound  can  be 
guaranteed  by  assuming  the  stabiiizaciii ty  of  the  system.  This  dees  not 
suffice  here  (except  for  scalar  x),  as  shown  in  the  following  example. 
Example  S:  Stabi 1 i z abi 1 i tv  not  sufficient  for  finite  cost 
Let  M  =2  where 


10  : 
1/2  : 


:  0 
:  0 


A 


2 


1/2  0  ; 

10  1/2  ; 


;  0 
:  0 


with  p^g  *  P21  *  1  and  P,j_  =  Pgj  B  0  "flip-flop  system  as  in  Figure  3). 
Both  forms  have  stable  dynamics  (eigenvalues  1/2,  1/2)  and  hence  are 

trivially  stabilizable.  However 


x 


k+2  " 


100.25 

5 


if  r. 
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'*  .V 


4 


*  f4  2fo  +  f2  f4 

0l'2)D(l)*S  14  i  £  * 

:  0  4 


D(1)D(2)S 


fl  f3  +  2f2 


fl  f4 


Both  Df 1)0(2)  and  D(2)D( 1)  have  4  as  an  eigenvalue.  Thus  grows  without 
bound  for  0  as  k  increases.  Controllabili  ty  in  each  form  allows  us 
to  place  the  eigenvalues  of  each  form's  closed  loop  dynamics  matrix  D(i)  as 
we  choose,  but  we  cannot  place  the  eigenvectors  arbitrarily.  In  this 
example  there  is  no  choice  of  feedback  laws  that  can  align  the 
eigenstructures  of  each  of  the  closed  loop  systems  so  that  the  overall 
dynamics  are  stable.  I 

The  following  example  demonstrates  that  (for  n  >.  2)  stabilizability  of 
even  one  form's  dynamics  is  not  necessary  for  the  costs  to  be  bounded. 
Sample  9;  Stabilizability  not  necessary  for  finite  cost 
Let  M  *  2  with 

A(l)  =  s  1  -Is  B(l>  *  :  0  : 

:  0  1/2  :  :  0  : 


A( 2)  =  :  1/2  1  :  8(2)  *  :  0  : 

:  0  1  :  :  0  : 

•  e  •  • 

Both  forms  are  unstable,  uncontrollable  systems  so  neither  is 
stabilizable.  We  again  take  the  form  dynamics  as  in  Figure  3. 
Then 


V  M 

x2k 

1  (A(2)A(l))k  x. 

|  ^ 

if  rQ  »  1 

1  (A(l)A(2))k  x g 

if  r0  »  2 
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V  V  V  V  V  V  V  V  V  V  */  \>  v  ••••*•*« 


'  A  . 


•Vv- 

-A 


A(1)A(2)  =  A(2)A(1) 


=  :  1/2  0 
:  0  1/2  s 

•  e 

Thus  x^ — >0,  and  hence  the  cost  is  finite.  We  next  show  that  this 
example  does  satisfy  Condition  1  of  Proposition  2.  From  (17)  with  F(l)  * 
F(2)  *  0  we  have 

Z(l)  =  Q(l)  +  A( 1) 'Z( 2)A( 1 ) 

Z(2)  =  Q(2)  +  A(2)'Z(1)A(2) 

Suppose,  for  convenience,  that  0(1)  *  0(2)  *  1.  Then  we  obtain  from  the 
first  equation  above  that 


2  '2' 

*  * 

Zi2(2)  i 

• 

i  +  z,1(2)  ~zn(2)  +(1/Z)Zi2(Z) 

• 

221<2) 

Z22(2)  : 

a 

• 

-Z,,(2)+(1/2)Z21(2)  1+Z11(2)-Z21(2)  i 

^  • 

• 

• 

« 

• 

a/4)Z22(2) 

•  a 

and  plugging 

this  into  the 

second  equation: 

’  2U(2) 

Z12(2)‘:  = 

• 

5/4  +(1/4)2, ,(2)  1/2  +(1/4)2, .(2)  : 

Za(2) 

a 

Z22(2)  / 

1/2  +(1/4)Z21(2)  3  +(1/4)Z22(2)  : 

•  a 

This  yields 

four  equations 

in  four  unknowns.  Solving  we  find 

'  Zu(l) 

zji<i)  * 

• 

:  6 

-14/3  : 

Z21(1) 

e 

z22a)  ; 

• 

•  a 

:  -14/3  13/3  : 

•  a 

and 

’  ZUC2) 

ZU(J,  | 

• 

a 

a 

a 

5  2/3  ’ : 

a 

Z21(2) 

Z2J<Z!  .  ■ 

a 

• 

2/3  4  : 

•  •  • 
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which  arc  both  positive  definite.  Thus  Zj  and  Z 2  satisfy  condition  (2)  of 
Proposition  2.  g 
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In  this  section  ut  examine  sufficient  conditions  for  the  existence  of 


finite  exoected  costs-to-go  that  replace  the  necessary  and  sufficient 
Conditions  1-3  in  Proposition  2,  and  are  somewhat  easier  to  compute,  in 
terms  of  the  spectral  norms  of  certain  matrices.  Recall  that  for  any 
matrix  A,  the  spectral  norm  of  A  is 

1/2 

1 1 A 1 1  *  max  < 1 1 Au I ) >  =  [max  eigenvalue(A'A) ] 


(lull  *  u'u  *  1  (25) 

Corollary  2;  Sufficient  conditions  for  the  existence  of  the  steady-state 
control  law  (and  finite  expected  costs-to-go)  for  the  time-invariant  JLQ 
problem  are  that  there  exist  a  set  of  feedback  control  laws 

VW  “  "F(Vxk 

such  that 


(1)  for  each  absorbing  form  i  (p  *  1),  the  pair 
(A(i),B(i))  is  stabilizable. 

(2)  for  each  recurrent  nonabsorbing  form  i  and  for  each 
transient  form  i  6  T  that  is  accessible  from 

a  form  j  6  C*i  in  its  cover  (j  *i): 

_2°.  t-i  to 

E  \  p..  ! i A( i ) -B( i )P( i ) ) '  !i  <  c  <  1  (26) 

/_  11 
t=l 


(3) 


for  each  transient  form  i  6  T  that  is  not  accessible  from 

tIt 

any  form  j  6  C  .  in  its  cover  (except  itself): 

-2-2.  »_i  t  ? 

E  \  oi4  I !A( i )-B( i )F( i ) j  I  r  <  oo 

/_ 

t«l 


(27) 


The  proof  of  this  Coroil*,,v  *s  immediate.  A  similar  result  for  continuous- 
time  systems  is  obtained  by  Honham  [3;Thm  6.1],  except  that  stabilizability 
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and  observabili ty  of  each  form  is  raquirad,  and  a  condition  lika  (26)  is 
required  for  all  nonabsorbing  forms. 

Condition  (2)  is  motivated  as  follows.  Tha  cost  incurred  while  in  a 
particular  transient  form  is  finite  wi th  orobabili tv  one  since,  eventually, 
the  form  process  leaves  the  transient  class  T  and  enters  a  closed 
communicating  class.  If  a  particular  transient  form  i  6  T  can  be  repeatedlv 
re-entered,  however,  the  expected  cost  incurred  while  in  i  may  be  infinite; 
(26)  excludes  such  cases.  Note  that  the  sufficient  conditions  of  Corollary 
2  are  violated  in  Example  3  (in  both  forms).  This  demonstrates  that  thev 
are  restrictive,  in  that  they  ignore  the  relative  "directions"  of  x  growth 
in  the  different  forms  (i.e.  the  eigenvector  structure).  We  consider  next 
a  sufficient  condition  that  is  easier  to  verify  than  Corollary  2.  but  is 
even  more  conservative. 

Corollary  3:  Sufficient  conditions  (l)-(2)  in  Corollary  2  can  be  replaced 
oy  the  following;  There  exists  a  set  of  feedback  control  laws 

U(W  *  "F(rk)xk 
such  that 

i i  (A(i)-B(i)F(i)  II  <  c  <  1  .  (23) 

The  proof  of  this  corollary  is  also  immediate.  0 

Note  that  if  (28)  holds  then  conditions  (l)-(3)  of  Proposition  2  hold. 
Note  also  that  we  are  guaranteed  that  lix^JI — >0  with  probability  one,  if 
(28)  holds  only  for  recurrent  forms.  However  this  is  not  enough  to  have 
finite  expected  cost,  as  demonstrated  in  the  following  examoles. 
fcUEEli  2l  Let 

•  a  a  a  a  a 

A( 1 )  «  ;  a  0  ;  A(l)  »  :  0  0  ;  8(1)  *  ;  0  :  »  B(2) 

:0a:  :  0  0  :  :  0  : 

a  a  a  a  a  a 

where  a  >  1,  and  with  Q(l)  *  I,  Q(2)  >  0,  Also,  let 
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where  a  >  1,  and  with  0(1) 

3  I . 

0(2) 

*  0,  Also, 

p  ?22 

*  1 

"12  *  P21 

*  0 

In  this  case 

1  1 

.  1  1 

min  | |A(1)-B(1)F(1)| |  * 

1  1 

a 

0  ill* 

F(  1 ) 

1  1 

:  0 

a  :!l 

1  1 

a 

.  II 

let 


a  >  1 


min  ||  A(2)-B(2)F<2)| I 


and  for  r^*! 


oo 


I  \ 
El  / 


,  ,-T  *'k  0(V\  +  °'k  "‘W 

I  k=0 


7  — ■ Si  3  k 
■  :ix0ll  \  (a?) 


/ 

k=0 


If  as  <  1,  then  the  expected  cost  i3 

2 


fixOi! 


<  oo 


a  o 


However,  .  if  a  p  >,1  then  the  expected  cost-to-go  is  infinite.  This 
demonstrates  that  (28)  holding  only  for  nontransient  forms  is  not 
sufficient  for  finite  expected  cost-to-go.  Soecif icallv,  as  this  example 
demonstrates,  the  cost-to-go  will  be  infinite  if  one  is  likely  to  remain 
sufficiently  long  in  transient  forms  that  are  unstable  enough.  0 

Example  10:  Let 


k+1 


1  1  : 
-1  -1  : 


if  rk  -  1,3 


x. 


k+1 


0  :  x, 


if  r  «  2 
k 


where  the  form  transition  dynamics  are  given  in  Figure  4.  Me  also  assume 


Discrete  Time  Markovian  JLQ  Optimal  Control 


Page  33 


4 


Q.  >  0,  i  ■  1,2,3. 

If  the  system  is  in  form  1  or  3  for  three  successive  times  (r.  »  r. 

k  k+1 

*  r^+2  *  *)*  th,n  xk+2  “  (®  ®)  for  anv  xk*  *n  *orm  r  *  2,  the  expe cted  cost 
incurred  until  the  system  leaves  (at  time  t)  given  that  the  state  at  time 
k  is  (xk,  rR  «  2)  is 

I— III  I  I  21  t  ~l 

E  I  \  x'  Q(2)x,  I  *  x'.  I  \  (A"  j  Q(2)A(2) 1  |x, 

\/_  I  I  /_  I  * 

l_  t»k  _!  I_  t«0  _l 

For  this  cost  to  be  finite  we  must  have 

_0£ 

/_  P„t(A'(2))t  Q(2)A(2)t  «  Q<2)  /__  p'a2'  <  oo 

t=0  “  t«0  22 

which  is  true  if  and  only  if 
2 

a  p22  <  1.  ( 29) 

Thus  we  would  expect  that  the  optimal  expected  costs-to-go  in  Proposition  2 
will  be  finite  if  and  only  if  (29)  holds.  We  next  verify  that  the  necessary 
and  sufficient  conditions  of  Proposition  2  say  this. 

The  matrix 

A(3)  *  8  1  1  : 

.•  -1  -1  : 

a  • 

is  nilpotentj  hence  the  absorbing  form  r  *  3  is  stabilizabie  (so  condition 
2  of  Proposition  2  is  met).  For  transient  forms  <1,2>  we  must  have 
0  <  5(1),  5(1)  <  oo  where 

_2£  _2£ 

3(1)  *  /_  P,,  1  A'(l)*  Of  1 )  A(l)t  +  /_  p..  t-1  A'(l)t  pt,G(2)A(l)t 
t«0  11  t«l 
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6(2)  »  /__  p*  A'(2;  Q( 2)  A(2r  *  Q(2)  /_  p  '  a‘l  *  - = 

t^o  22  t*o  i  -  ©22  a 

Thus  for  6(2)  to  be  positive  definite  we  have  the  condition  (29).  Finally 
since  A(l)f  *  0  for  t  2  2,  we  have 

6(1)  *  Q(l)  +  A'(l)  tp11Q(l)  +  P12Q(2>]  A(l) 

p. -0(2) 

*  Q(  1 )  +  A'  (1 )  [  p. , Q(  1 )  + - - - 5 - ]  A(l) 

1  -  P22*2 

which  is  positive-definite  since  Q(l),  Q(2)  >  0.  Thus  the  necessary  and 

sufficient  conditions  of  Proposition  2  here  reduce  to  (29).  Note  that  he 

sufficient  condition  (28)  of  Corollary  3  is  never  met  for  r  =  1  and  r  *  3, 

since  ||A(1)|!  =  ||A(3)1I  »  2,  and  to  meet  (28)  for  r  *  2  requires  lai  <  1. 
On  the  other  hand,  the  sufficient  conditions  for  Corollary  2  are  met  if 
(29)  holds  because  forms  <1,2>  are  'non-re-enterabie'  transient  forms 
satisfying  (27) .  ] 
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7.  Summary 

In  this  paper  we  have  formulated  and  solved  the  discrete-time  linear 
quadratic  control  problem  with  perfect  observations  when  the  system  and 
cost  parameters  jump  randomly  according  to  a  finite  Harkov  process.  The 
optimal  control  law  is  linear  in  at  each  time  k,  and  is  different  (in 
general)  for  each  possible  set  of  parameter  values.  Proposition  2  provides 
necessary  and  sufficient  conditions  for  existence  of  the  optimal  steady- 
state  JIQ  controller.  These  conditions  are  not  easily  tested,  however, 
since  they  require  the  simultaneous  solution  of  coupled  matrix  equations 
containing  infinite  sums  .  In  Corollaries  2  and  3,  sufficient  conditions 
3re  presented  that  are  more  easily  tested. 

Perhaos  the  most  imoortant  contribution  of  this  paper  is  the  set  of 
examples  that  explore  the  reasons  for  the  complexity  of  the  conditions  of 
Proposition  2.  For  example  we  have  shown  that  stabilicabili ty  of  the 
system  in  each  form  is  neither  necessary  nor  sufficient  for  the  existence 
of  a  stable  steady-state  closed-looo  system.  Issues  such  as  the  amount  of 
time  spent  in  unstable  forms,  and  the  differences  among  the  stable  and 
unstable  subspaces  in  different  forms  have  been  illustrated. 
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